Turing Machines 


Part Iwo 


Recap from Last Time 


The Turing Machine 


e A Turing machine consists of three parts: 
- A finite-state control that issues commands, 


- an infinite tape for input and scratch space, 
and 


- a tape head that can read and write a single 
tape cell. 


e At each step, the Turing machine 


- writes a symbol to the tape cell under the tape 
head, 


- changes state, and 


- moves the tape head to the left or to the right. 


Input and Tape Alphabets 


s A Turing machine has two alphabets: 


- An input alphabet 2. All input strings are written 
in the input alphabet. 


- A tape alphabet I, where È C I’. The tape alphabet 
contains all symbols that can be written onto the 
tape. 


e The tape alphabet I can contain any number of 
symbols, but always contains at least one blank 
symbol, denoted Ll. You are guaranteed O ¢ È. 


e At startup, the Turing machine begins with an 
infinite tape of [| symbols with the input written at 
some location. The tape head is positioned at the 
Start of the input. 


A Simple Turing Machine 


Each transition of the form 


x>y,D 


means “upon reading X, replace it with 

symbol y and move the tape head in 

direction D (which is either L or R). 

The symbol L] represents the blank 
symbol, 


This special accept 
state causes the 
machine to 
immediately accept. 


This special reject 
state causes The 
machine to 


immediately reject. 
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New Stuff! 


Main Question for Today: 
Just how powerful are Turing machines? 


A Sample Algorithmic Problem 


Composite Numbers 


s A natural number n = 2 is composite if it is 


the product of two natural numbers p and q, 
neither of which is n. 

Example: 

- 6 =3:21s composite. 

- 42 = 7 ; 6 is composite 

- 5 is not composite. 

Programming problem: write a method that 


determines whether a natural number is 
composite. 


bool isComposite(int n) { 
if (n < 1) return false; 


int divisor = 2; 
while (divisor z n) 1 
if (n is a multiple of divisor) { 
return true; 


divisor++; 


} 


return false; 


Question: Can we build a TM that 
determines whether a number is 
composite? 


Back to Languages 


e Let = {1} and consider the following 
language L over 2: 


L = { 17 | nis composite } 
s This language is not regular (think about 
why this is). 
e It's also not context-free (don't worry if you 
don't see why - we didn't discuss this. ©) 


e Can we build a TM for it? 


(eo) Gee 1—1,L 


bool isComposite(int n) { 
if (n < 1) return false; 


int divisor = 2; 
while (divisor z n) 1 
if (n is a multiple of divisor) { 
return true; 


} 


divisor++; 


return false; 


int divisor = 2; 
while (divisor z n) 1 
if (n is a multiple of divisor) { 
return true; 


divisor++; 


Reusing our IMs 


s What does our TM need to be able to do? 
- Check if divisor =n 
- Check ifn is a multiple of divisor. 

s What TMs did we build last time? 
- One that checks for 071 


- One that checks for 0°17, where m is a 
multiple of n. 


s Coincidence? I think not! 


A Sketch of the TM 


+ 
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bool isComposite(int n) { 
if (n < 1) return false; 


"int divisor = 2; Ñ 
AL Té CULV tsar en) A? 
if (n is a multiple of divisor) { 
return true; 


} 


divisor++; 


return false; 


A Sketch of the TM 


+ 


00/11/11 1 


bool isComposite(int n) { 
if (n < 1) return false; 


while ss 2 re ae 
ten tS "FT mutttple SF divisor) { 
return true; 


} 


divisor++; 


Somehow, run our TM from 
last time to see it the string 
is ot the form om", 
return false; 


A Sketch of the TM 


T 


00/11/11 1 


bool isComposite(int n) { 
if (n < 1) return false; 


int divisor = 2; 


while (divisar. E E 
fif (n is a multiple of divisor) L 
ı return true; 


U 
LB man leda m m m m m m Somehow, run Our TM from 


divisor++; 
last time to see if the string 


is ot the form om", where n 
return false; is a multiple of m. 


How can we get our new IM 
to run our old TMs? 


Function Call and Return 


e Think about actual programming for a 
minute. 


e If you write a function that performs a 
task, another part of the code can call 
that function to use the functionality. 


e Could we do something like that here in 
our IM? 


Function Call and Return 


e Internally, the memory 
for each function call 
is allocated on the 
Stack: 


- Each call allocates 
more space. 


- Each return cleans up 
the space. 


- Arguments are copied 
by value. 


Combining TMs 


e Claim: We can use the TM's tape to 
Simulate function call and return. 


e This means that we can build very 
complex IMs by assembling smaller 
subroutines and combining them 
together. 


e The details are icky; we'll see them in a 
second. 


Our IM, in More Depth 


: oo11111 |- 


High-level idea: Pass 
the current string as 


a “parameter” to the 
TM for 0"1”. 


Our IM, in More Depth 
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High-level idea: Pass 
the current string as 


a “parameter” to the 
TM for 0"1”. 


Our IM, in More Depth 


oo11111 


Now, go run the TM for 0"1" by 
transitioning into its start state. It 
never takes more than one step off 


the string in either direction, so it 
has no idea there's anything else 
on the tape. 


Our IM, in More Depth 


The TM would normally reject 


here. But we can change that! 
Suppose we change the TM's 
reject state into a work state that 
cleans everything up. 


Our IM, in More Depth 
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High-level idea: 
Clean up the stack 


Space to return from 
the function call. 


Our IM, in More Depth 


~ oo11111 |- 


High-level idea: 
Clean up the stack 


space to return from 
the function call. 


Our IM, in More Depth 


~ oo11111 |- 


Now, you can 
imagine that we do 
the same thing, but 

for the TM that 


checks ifthe number 
of 1's is a multiple of 
the number of 0's. 


Subroutines in TMs 


e Just as complex programs are often broken 
down into smaller functions and classes, 
complex TMs are often broken down into 
smaller “subroutines.” 


e Each subroutine performs some task that 
helps in the overall task. 


e The TM is then described by giving a 


collection of subroutines and showing how 
they link up. 


The “Copy” Subroutine 


e This subroutine starts with the tape head at the 
start of a string of Os and 1s: 


+ 


00/1 1/11 1 


e It ends in this configuration: 


T 


x 0 0711111 x 0 O0 1 1 1 


Im 
Im 


e We use the copy subroutine to let us run 
another IM on the current input without 
breaking it. 


The “Cleanup” Subroutine 


e This subroutine starts with the tape head 
between two * characters delimiting TM 
workspace: 


T 


ok 1x x 00 x x 0 


lo 
Im 
Im 
Im 
Im 
Im 


e It ends in this configuration: 


T 


00/11/11 1 


e We use the cleanup subroutine to recover from 
the end of running a sub-TM. 
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TM Subroutines 


e To represent a subroutine call inside a 
TM, just add a state with dashed edges 
with the name of the subroutine. 


e If the subroutine is just a “processing” 
subroutine, have a single exit arrow. 


e If the subroutine may exit along 
different paths, have multiple exit 
arrows. 


Defining a Subroutine 


e If you want to (or are asked to) design a 
TM subroutine, design it like a normal 
TM with one change: have special 
dashed states representing the exit of 
the subroutine. 


Time-Out for Announcements! 


Second Midterm Exam 


e The second midterm exam is next Monday, November 16 from 
/PM - 10PM. 

e Room assignments divvied up by last (family) name: 
- Aga - Ven: Go to Hewlett 200. 
- Ver - Zhe: Go to 370-370 


e Cumulative exam with a focus on material from Lectures 10 - 18 
and PS4 - 6. 


- Main focus is on graphs, the pigeonhole principle, induction, finite 
automata, regular expressions, and regular languages. 
- Will not test the Myhill-Nerode theorem, CFGs, or Turing machines. 


- Since everything builds on itself in this course, the exam may require 
topics from earlier in the course. 


e As before, you get one double-sided, 8.5” x 11” sheet of notes 
during the exam. 


Exam Practice 


We have a practice midterm tonight 
7PM - 10PM in room 320-105. 

- Can't make it? Don't worry! We'll release 
electronic copies of the exam this evening. 

e We're releasing practice problems 
throughout the week. Please ping us if 
you have any specific topics you'd like to 
request! 


Solution Sets 


e Solution sets for the problem sets and 
practice problems are available for pickup 
in Gates. 

e We're running out of filing cabinet space. 
We are planning on recycling all unclaimed 
solutions sets for topics prior to the exam 
later this week. 


e Claim them now or forever hold your 
peace! 


Your Questions! 


“I feel like since the midterm was so "easy" 
(as in the 75th percentile was so high), the 
exam tested one's carefulness more than 
one's understanding of and ability to apply 
ideas from the course (i.e., trivial mistakes 
were costly). Why am I wrong?” 


I read over a bunch of exams (100+) when putting the criteria together 
and tried to put together a set of criteria that didn't penalize people 
for minor mistakes. If you feel like we were deducting a lot for trivial 
mistakes, come talk to me about it - I suspect that you may have had a 
deeper misunderstanding that looks like a really nitpicky deduction. 


I don't think that this test was by any stretch “easy” or just a test of 

caretulness. This was a hard test. A lot of people did do quite well, 

but there was a very long tail and a lot of people struggled. Having 

talked with a lot of people one-on-one about this, my sense is that it 

pushed people hard on concepts, on manipulating detinitions, on nuances 
of the techniques that we've seen so far, etc. 


“We've talked about the Duck Syndrome 
and other pressures that influence the 
student culture at Stanford. I was 
wondering what the culture of being a 
professor at Stanford is like. How is it 
Similar and different from being a student 
here?” 


My experience as a lecturer is pretty 
different trom the traditional tenure-track 
protessor vole. 


Other Than that, I'm going To save this 
answer tor in class. © 


“I've done some internship interviews but I 
struggle technical questions, even if they're 
stuff I learned in 106B. I'm doing alright in 
my CS classes, but it just seems like I can't 
Slap what I've learned onto the real world. 
Tips?” 
A lot of this is Just practice and repetition. You never realize how 


much you don't know until you need to do it and realize that There 
are a bunch of concepts you don't fully understand. 


My recommendation is to work through a bunch of practice problems 
Whenever you're struggling, try to identity why. Is there some 


concept you don't fully get? Is there a tool you don't actually really 
know how To use? 


Also, check out “Cracking the Coding Interview’ and work through 
some ot the practice problems we've released in CS4 
(http://cs9.stanford.edu ) 


Back to CS103! 


Main Question for Today: 
Just how powerful are Turing machines? 


How Powerful are TMs? 


e Regular languages, intuitively, are as 
powerful as computers with finite 
memory. 


s TMs by themselves seem like they can do 
a fair number of tasks, but it's unclear 
specifically what they can do. 


e Let's explore their expressive power. 


Claim 1: Computers with unbounded 
memory can simulate Turing machines. 


“Anything that can be done with a TM 
can also be done with an unbounded- 
memory computer.” 


Go 


qı 


q2 


q3 


The TM's finite-state control 
can be encoded as a table, 
making it easy for a 
computer to look up 
transitions information. 


Simulating a IM 


e To simulate a TM, the computer would need to be 
able to keep track of 


e the finite-state control, 
e the current state, 


e the position of the tape head, and 
e the tape contents. 


e The tape contents are infinite, but that's because 
there are infinitely many blanks on both sides. 


e We only need to store the “interesting” part of the 


tape (the parts that have been read from or written 
to so far.) 


Claim 2: Turing machines can simulate 
computers with unbounded memory. 


“Anything that can be done with an 
unbounded-memory computer can be done 
with a TM.” 


What We've Seen 


e [Ms can 


implement loops (basically, every TM we've seen). 
make function calls (subroutines). 


keep track of natural numbers (written in unary on 
the tape). 

perform elementary arithmetic (equality testing, 
multiplication, addition, division, etc.). 


perform if/else tests (different transitions based on 
different cases). 


What Else Can TMs Do? 


e Maintain variables. 


s Have a dedicated part of the tape where the 
variables are stored. 


e Each variable can be represented as a name (written 
one symbol at a time) followed by a value. 


e Maintain arrays and linked structures. 


s Divide the tape into different regions corresponding 
to memory locations. 


e Represent arrays and linked structures by keeping 
track of the ID of one of those regions. 


A CS107 Perspective 


e Internally, computers execute by using basic 
operations like 


e simple arithmetic, 
e memory reads and writes, 
e branches and jumps, 


e register operations, 
° etc. 


e Each of these are simple enough that they 
could be simulated by a Turing machine. 


A Leap of Faith 


s It may require a leap of faith, but anything you 
can do a computer (excluding randomness and 
user input) can be performed by a Turing 
machine. 


e The resulting TM might be colossal, or really 
Slow, or both, but it would still faithfully simulate 
the computer. 


e We're going to take this as an article of faith in 
CS103. If you curious for more details, come talk 
to me after class. 


Just how powerful are Turing machines? 


Effective Computation 


s An effective method of computation is a form 
of computation with the following properties: 
e The computation consists of a set of steps. 


e There are fixed rules governing how one step leads to 
the next. 


e Any computation that yields an answer does so in 
finitely many steps. 


s Any computation that yields an answer always yields 
the correct answer. 
e This is not a formal definition. Rather, it's a set 


of properties we expect out of a computational 
system. 


The Church-Iuring Thesis claims that 


every effective method of computation is either 
equivalent to or weaker than a Turing machine. 


“This is not a theorem - itis a 
falsifiable scientific hypothesis. 
And it has been thoroughly 
tested!” 

- Ryan Williams 


CFLs 
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Problems 
Solvable by 
Any Feasible 
Computing 
Machine 


All Languages 
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Problems 
solvable by 
Turing 
Machines 


All Languages 


IMs = Computers 


e Because Turing machines have the same 
computational powers as regular computers, we 
can (essentially) reason about Turing machines 
by reasoning about actual computer programs. 


e Going forward, we're going to switch back and 
forth between TMs and computer programs 
based on whatever is most appropriate. 


« In fact, our eventual proofs about the existence of 
impossible problems will involve a good amount 
of pseudocode. Stay tuned for details! 


Strings, Languages, 
Encodings, and Problems 


What problems can we solve with a computer? 
A 


What kind of 
computer? 


What problems can we solve with a computer? 
N 


What is a 
‘problem?’ 


Languages and Problems 


e We've been using formal languages as a way 
of modeling computational problems. 


e However, the problems we encounter in The 
Real World don't look at all like language 
problems. 


e Is this all theoretical nonsense? Or is there a 
reason for this? 


“In theory, there's no difference between 
theory and practice. In practice, there is.” 


Decision Problems 


e A decision problem is a type of problem where the 
goal is to provide a yes or no answer. 


e Example: checking arithmetic. 
Given x, y, and z, is x+y=z? 
e Example: detecting relationships. 


Given a family tree T and people x and y, 
is x a grandparent of y? 


e Example: avoiding traffic. 


Given a transportation grid G annotated with traffic 
information, a start location s, a destination d, and a time 
limit t, is there a way to get from s to d within time t? 


Solving Decision Problems 


Solving Decision Problems 


input 


How do we 
represent our 
inputs? 


Strings and Objects 


« Think about how my 
computer encodes the 
image on the right. 


Internally, it's justa 
series of zeros and 
ones sitting on my 
hard drive. 


All data on my 
computer can be 
thought of as (suitably- 
encoded) strings of Os 
and 1s. 


Strings and Objects 


e A different sequence of 
Os and 1s gives rise to 
the image on the right. 


e Every image can be 
encoded as a sequence 
of Os and 1s, though 
not all sequences of Os 
and 1s correspond to 
images. 


Strings and Objects 


Let Obj be some discrete, finite object (a string, a 
video, an image, a text file, etc.) 


Let X be some alphabet. 


We'll represent an encoding of Obj using the 
characters in È by writing (Obj). Think of (Obj) like a 
file on disk - it encodes complex data as a series of 
characters. 


A few remarks about encodings: 


e We don't care how we encode the object, just that we can. 


e The particular choice of alphabet isn't important. Given any 
alphabet, we can always find a way of encoding things. 


e We'll assume we can perform “reasonable” operations on 
encoded objects. 


Strings and Objects 


e Given a group of objects Objı, Objz, ..., Objn, we 
can create a single string encoding all these 
objects. 

e Think of it like a .zip file, but without the 
compression. 

e We'll denote the encoding of all of these objects 
as a single string by (Objı, ..., Objn). 


e This lets us feed multiple inputs into our 
computational device at the same time. 


Solving Decision Problems 


input 
(some string) 


How do we specify 
the behavior we 
want? 


Specifying a Decision Problem 


e Consider this decision problem: 


Given x, y, z € N, determine 
whether x+y=z. 


e With our computational model, we'll feed some 
string into a TM, and it then might come back 
with an answer (yes or no). 


e Some strings are accepted, some are rejected, 
and some cause the machine to loop infinitely. 


Specifying a Decision Problem 


e Consider this decision problem: 


Given x, y, z € N, determine 
whether x+y=z. 


e If we give the input as (x, y, z), the set of 
strings the TM should say YES to is 


{ (xyz |x y zEeNandx+y=z} 


e Notice that this is a language - it's a set of 
strings! 


Specifying a Decision Problem 


e Consider this decision problem: 
Given a graph G, determine whether 
G is a bipartite graph. 


e With our computational model, we'll feed some 
string into a TM, and it then might come back 
with an answer (yes or no). 


e Some strings are accepted, some are rejected, 
and some cause the machine to loop infinitely. 


Specifying a Decision Problem 


e Consider this decision problem: 


Given a graph G, determine whether 
G is a bipartite graph. 


e If we give the input as (G), the set of strings 
the TM should say YES to is 


{ (G) | Gis a bipartite graph } 


e Notice that this is a language - it's a set of 
strings! 


Problems and Languages 


e Key intuition: Every language 
corresponds to some decision problem. 
e Example: 
s { (x, y) |x, y eN and x =s; y } is a language. 
e It corresponds to the following decision 
problem: 
Given x, y € N, do x and y leave the 
Same remainder when divided by 3? 


Problems and Languages 


e Key intuition: Every language 
corresponds to some decision problem. 
e Example: 


e { (D) | Dis a DFA that accepts £ }isa 
language. 


e It corresponds to the following decision 
problem: 


Given a DFA D, does D accept £? 


Problems and Languages 


e Key intuition: Every language 
corresponds to some decision problem. 
e Example: 
e { (G) | Gis a planar graph } is a language. 


e It corresponds to the following decision 
problem: 


Given a graph G, is G planar? 


Problems and Languages 


e Key intuition: Every decision problem on 
finite, discrete objects corresponds to some 
language. 


s If II is a problem and the inputs to II are 
objects x1, X2, ..., Xn, we can form the language 


{ (x1, X2, ..., Xn) | the answer to problem 
II, given inputs xı, ..., Xn, 
is “yes.” } 


What All This Means 


e Our goal is to speak of computers solving 
problems. 


e We will model this by looking at TMs 
recognizing langudges. 


e For decision problems that we're 
interested in solving, this precisely 
captures what we're interested in 
capturing. 


